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Executive Summary 


Background. The Minnesota Principals Academy (MPA) is an 18-month executive development program 
designed to enhance the performance of school leaders and improve educational outcomes for students. MPA 
uses the National Institute for School Leadership (NISL) executive development program, to provide principals 
with research-based information that they can use in their schools. 


The purpose of this evaluation was to describe and understand the MPA as a resource and training program for 
school and district leaders. Our evaluation is designed to answer the following questions: 
1. To what extent have participants gained knowledge in the areas covered by the Minnesota Principals 
Academy (NISL) curriculum? 
2. To what extent do the supervisors of Minnesota Principals Academy participants see a change in 
participants’ knowledge in the areas covered by the Minnesota Principals Academy (NISL) curriculum? 
3. To what extent have student outcomes (e.g., scores on statewide reading and math tests) in 
participants’ schools changed as a result of participation in the Minnesota Principals Academy? 


This report provides information on the first two evaluation questions for the two cohorts who just completed 
the program. The third evaluation question cannot be addressed until after the participants have had time to 
make changes in their practice (e.g., it may be useful to analyze test scores after spring 2018 statewide testing). 
The Northwest (NW) Cohort began their work in July 2015; the Twin Cities (TC) Cohort began in October 2015. 
Both groups completed the MPA in June 2017. 


Methods. To answer the question about the knowledge gained by participants in the areas covered by the MPA 
curriculum, participants from both cohorts were asked to complete an online survey at the beginning of their 
participation in the program (Fall 2015) and upon completion of their participation in the program (April 2017). 
The online survey was designed to obtain information about participants’ knowledge, skills, and behaviors on a 
variety of topics covered in the MPA. The purpose of administering a pre-survey and a post-survey was to 
examine changes in the participants from the beginning of the program to the end of the program; thus, the 
information presented in this report is only for the 45 participants who completed both the pre- and post- 
surveys (18 from the NW Cohort and 27 from the TC Cohort). 


To answer the question relating to the extent to which the supervisors of MPA participants see a change in 
participants’ knowledge in the areas covered by the MPA curriculum, participants’ school/district supervisors 
were also surveyed. The online survey was sent to 33 supervisors in April 2017, some of whom supervised more 
than one participant. A total of 17 (52%) supervisors responded to the survey (12 from the TC Cohort and 5 from 
NW Cohort). The 17 supervisors who responded represented 34 of the 53 participants (20 of the NW Cohort 
participants and 14 of the TC Cohort participants). For both the participant and supervisor surveys, the results 
from both cohorts are reported as combined data. 


Survey Results. In general, the results from both the participant and supervisor surveys suggest a positive 
change in participants’ knowledge, skills, and behaviors in the areas covered by the MPA. In addition, all of the 
participant respondents said they would recommend the MPA to a colleague, and 94% of the supervisor 
respondents said they would be likely or very likely to send additional members of their staff to the MPA. 


On the supervisor survey, respondents were asked to rate their level of agreement with 12 statements related 
to changes they have seen in their staff members as a result of participating in the MPA. For example, the 


Center for Applied Research and Educational Improvement, University of Minnesota 1 


respondents were asked to rate their level of agreement with the statement “The participant(s) has enhanced 
his/her leadership skills.” Other topical areas included rating whether participants had changed, as a result of 
participating in the MPA, their ability to motivate and support teachers, especially related to new instructional 
practices; their thinking around how to improve instruction and build teams; their willingness and ability to 
ensure that they and their teachers use data and research to make curricular and instructional decisions; their 
commitment to high expectations for all students; and their willingness to listen to a variety of stakeholders. The 
survey results were extremely positive. In fact, over 85% of the supervisor respondents agreed or strongly 
agreed with all of the 12 statements regarding Academy participation. Their comments also suggested that both 
the participants and their schools and districts had benefitted as a result of the MPA. 


On the pre- and post-participant surveys, respondents were first asked to rate their level of expertise in a variety 
of areas. Respondents’ ratings of their level expertise in these areas increased from the pre- to the post-survey. 
The areas where respondents reported the greatest increase in expertise were: 
e Developing teachers as leaders (27% increase from 35% to 62% good); and 
e Solving problems systematically by examining the whole picture, rather than isolated parts (25% 
increase from 39% to 64% good). 


Respondents were also asked to indicate their level of agreement to statements regarding instructional 
practices, standards, and teacher performance and student outcomes. For these areas, the statements where 
agreement increased the greatest after participating in the Academy were: 


Instructional Practice: 
el insist that the way to improve student learning is by improving instructional practices (22% increase 
from 24% to 46% strongly agree); and 
e | encourage and support innovative improvement practices (17% increase from 32% to 49% strongly 
agree). 


Standards: 
e | convey to teachers how important it is for classroom assessments to be carefully aligned to content 
standards (25% increase from 67% to 92% agree/strongly agree); 
e | convey to teachers how important it is for classroom assessments to be carefully aligned to 
performance standards (20% increase from 64% to 84% agree/strongly agree); and 
e lam comfortable defining for teachers what constitutes effective standards for instructional practice 
(16% increase from 84% to 100% agree/strongly agree). 


Teacher Performance and Student Outcomes: 

e | discuss why student depth of understanding is critical to achieving our mission (31% increase from 55% 
to 86% agree/strongly agree); 

e | search widely to benchmark our school’s performance to identify strategies that have enabled high 
performing schools to sustain improvement (21% increase from 41% to 62% agree/strongly agree); 

e [have institutionalized processes that ensure the professional development system is aligned with a 
belief that all students can achieve the same high standards (18% increase from 65% to 83% 
agree/strongly agree); 

e |have established a clear, written statement of learning and teaching principles for teachers to guide 
their improvement efforts (17% increase from 43% to 60% agree/strongly agree); and 
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Responses to the survey items related to how frequently Academy participants interact with teachers in various 
situations all increased in frequency from the pre- to the post-survey with the exception of “modeling 
instructional strategies for teachers” in which over one-third (35%) of respondents consistently reported they do 
never or seldom. Over three-fourths of respondents said they do the following often or almost always: 

e give teachers specific ideas for how to improve their instruction (75% often, 8% almost always); 

e use research evidence in decision-making (57% often, 22% almost always); 

e discuss instructional issues with your teachers (62% often, 14% almost always); 

e visit classrooms to briefly observe instruction (51% often, 27% almost always); and 

e Provide or locate resources to help staff improve their teaching (59% often, 16% almost always). 


Academy participants were asked a series of questions about practices in their districts, their schools, and by 
their teachers. Although levels of agreement generally went up in all of these areas, there were a few 
statements where the shift in agreement level was toward greater disagreement. This occurred with four 
statements relating to district practices; the district statement where the shift was most notable from the pre- 
to the post- survey was: 
e Instructional leadership is a key component of our principal evaluation system (13% decrease from 91% 
to 78% agree/strongly agree). 


The level of agreement reported by respondents with statements about school-level practices increased overall. 
The statements where agreement increased by almost 20% or more between the pre-and post-survey are listed 
below: 
e We provide an aligned curriculum for students across the grades (38% increase from 43% to 81% agree/ 
strongly agree); and 
e There is a strong commitment in this school to a common set of shared goals (19% increase from 70% to 
89% agree/ strongly agree). 


In general, the agreement levels with statements relating to teacher practice increased; however, the 
agreement levels started higher on the pre-survey therefore the increases were not as large as in some of the 
other areas covered on the survey. The statements with greatest increase in agreement levels expressed by 
respondents were: 
e Teachers in this school have a sense of collective responsibility for student learning (14% increase from 
73% to 87% agree/strongly agree); and 
e Teachers in this school view problems as issues to be solved, not as barriers to action (12% increase 
from 52% to 64% agree/strongly agree). 


Academy participants were asked on both the pre-and the post-surveys to report what percentage of their time 
in an average week they spend on the following activities: instructional leadership, public engagement, and 
organizational management. The average amount of time reported remained fairly consistent between the two 
surveys (approximately 38% on instructional leadership, 13% on public engagement, and 49% on organizational 
management). Note, however, that the range of responses was quite large. For example, on the pre-survey, 
respondents reported spending anywhere from 10% to 85% on instructional leadership (10% to 70% on post- 
survey), 5% to 40% on public engagement (0% to 60% on post-survey), and 10% to 85% on organizational 
management (15% to 85% on post-survey). 


Center for Applied Research and Educational Improvement, University of Minnesota 3 


Respondents were asked to share comments on what they considered the greatest take-away from the MPA. 
The most common response from participants focused on the importance of leadership. Seventeen participants 
discussed the importance of being an instructional leader and possessing strong leadership skills. In line with 
this, many respondents also said that the most helpful content (if they had to pick just one) was learning more 
about instructional leadership and coaching in the content areas (math, science, literacy, and social studies). In 
keeping with this theme, when respondents were asked to discuss how their practice had changed as a result of 
participation in the Academy, they overwhelmingly talked broadly about changes in their leadership practice 
(e.g., restructuring leadership team meetings, driving change within their buildings) and taking on the role of 
instructional leader within their schools. 


Respondents also provided feedback on the program. Most said it was a positive experience and had no 
suggestions for program improvements. Many said they enjoyed the activities and content presented during 
cohort meetings as well as the opportunity for networking with other professionals, but some wanted even 
more time to learn from the other cohort members. Some shared various suggestions for improving the content 
(e.g., changing the EL unit or providing additional information around MTSS); making changes to the Action 
Learning Project (e.g., getting rid of it or letting people decide it what they would work on at a later point in 
time); or changing the timing of the program (e.g., reducing the number of days out of the building). 


In summary, looking across all of the survey items and comments a main finding emerges — the Academy 
participants are thinking more about leadership and the role of an instructional leader. The participants are 
noticing this change and so are their supervisors. Participants now know more about that concept and are trying 
to determine how that role will fit within their jobs, their schools, and their districts. Most importantly, the 
survey results suggest that participants are making changes in key areas identified by the MPA/NISL curriculum. 
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Introduction 


Program Overview 


The Minnesota Principals Academy (MPA) is an 18-month executive development program that provides 
ongoing professional development to enhance the performance of school leaders and improve educational 
outcomes for students. The University of Minnesota’s Department of Organizational Leadership, Policy, and 
Development (OLPD) in the College of Education and Human Development hosts the program, with financial 
support provided by the Minnesota Legislature and participants’ school districts. 


The MPA uses the National Institute for School Leadership (NISL) executive development program to provide 
principals with research-based information that they can use in their schools.* The NISL curriculum includes 13 
units designed to help participants in three areas that are key to their role: leadership skills, best practices in 
standards-based education, and content area instruction. University of Minnesota faculty also enhance the 
content of the NISL curriculum by providing additional information in areas that are of interest and importance 
for participants. Over the course of the 18-month program, participants are provided with approximately 30 
days of face-to-face instruction offered in two-day segments with their cohort, 40 hours of online content, and 
readings. All participants also complete an Action Learning Project intended to encourage participants to 
examine an instructional issue or area relevant to their school. In addition, the Action Learning Project provides 
a framework that can be used after program completion. 


Evaluation Purpose 


The purpose of this evaluation was to describe and understand the Minnesota Principals Academy as a resource 
and training program for school and district leaders. Our evaluation was designed to answer the following 
questions: 
1. To what extent have participants gained knowledge in the areas covered by the Minnesota Principals 
Academy (NISL) curriculum? 
2. To what extent do the supervisors of Minnesota Principals Academy participants see a change in 
participants’ knowledge in the areas covered by the Minnesota Principals Academy (NISL) curriculum? 
3. To what extent have student outcomes (e.g., scores on statewide reading and math tests) in 
participants’ schools changed as a result of participation in the Minnesota Principals Academy? 


This report provides information on the first two evaluation questions for the two cohorts who just completed 
the program. The third evaluation question cannot be addressed until after the participants have had time to 
make changes in their practice (e.g., it may be useful to analyze test scores after spring 2018 statewide testing). 
The Northwest (NW) Cohort began their work in July 2015; the Twin Cities (TC) Cohort began in October 2015. 
Both groups completed the MPA in June 2017. 


1 For more information on NISL, please visit their website: http://www.nisl.net/. 
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Evaluation Question 1: Changes in Participants 


This section describes the methods used, and results obtained, for Evaluation Question 1: “To what extent have 
participants gained knowledge in the areas covered by the Minnesota Principals Academy (NISL) curriculum?” 


Methods 


To answer the evaluation question about the extent to which participants gained knowledge in the areas 
covered by the MPA, participants from both cohorts were asked to complete an online survey at the beginning 
of their participation in the program (Fall 2015) and upon completion of their participation in the program (April 
2017).”? The survey was designed to obtain information about participants’ knowledge, skills, and behaviors on 
a variety of topics covered in the MPA. The purpose of administering a pre-survey and a post-survey was to 
examine changes in the participants from the beginning of the program to the end of the program. 


Both the pre- and post-surveys were administered using an online survey tool. Participants received an initial 
email inviting them to complete the survey; several reminders were also sent via email to those who had not yet 
completed the survey. Participants were asked to respond to statements about instructional leadership, 
communication of goals, standards-based alignment, distributed leadership, and data-based decision-making; in 
addition, participants were asked to provide demographic information, including their current position, length of 
time in their current position, and whether or not they were in the same position as at the beginning of the 
program. Respondents working in a district-level position and not in a school building were asked to only 
complete the statements about their districts; all other participants completed both district-level and building- 
level questions. 


Pre-survey. The pre-survey consisted of two demographic items, sixty-nine closed-ended items, and one open- 
ended item. All 62 Minnesota Principals Academy participants were invited to complete the pre-survey; overall, 
56 completed the pre-survey for a response rate of 90%. From the NW Cohort, 24 of the 27 participants 
completed the survey for a response rate of 89%. From the TC Cohort, 32 out of 35 completed the survey for a 
response rate of 91%. 


Post-survey. The post-survey consisted of three demographic items, sixty-eight closed-ended items, one item 
asking for a yes/no answer and short explanation, and five open-ended items.* The post-survey was 
administered to 53 remaining participants? in the Academy at the end of the program and had a total response 
rate of 91% (n=48). From the NW Cohort, 20 of the 23 participants completed the survey for a response rate of 
87%. From the TC Cohort, 28 out of 30 completed the survey for a response rate of 93%. 


? Respondents were asked to complete the survey at the end of the April meeting rather than during the final June meeting, 
because the June meeting was devoted largely to the presentations of findings from participants’ Action Learning Projects. 
3 In addition to completing the pre- and post-program surveys, participants were asked to complete a retrospective pre- 
survey in spring 2017. The purpose of administering the retrospective pre-survey was to compare survey designs for 
collecting pre/post data from Academy participants (see Appendix for more information on the retrospective pre-survey). 
Based on our analysis, a decision was made to report only the traditional pre/post-survey data here. 

“ The post-survey consisted of the same closed-ended items as on the pre-survey; however, one item from the pre-survey 
was inadvertently left off the post-survey; thus, there were only sixty-eight items on the post-survey. Also, an additional 
demographic item was added to the post-survey asking whether or not the participant held the same position as at the 
start of the Academy. 

5 Some participants left the program for a variety of reasons (e.g., retirement) or were not asked to complete the survey 
because they were not working in a school or district role. 
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Analysis. As mentioned above, the purpose of administering a pre-survey and a post-survey was to examine 
changes in the participants from the beginning of the program to the end of the program; thus, the information 
presented here is only for those participants who completed both the pre- and post-surveys. Forty-five 
participants completed both the pre- and post-surveys (18 from the NW Cohort and 27 from the TC Cohort). 
Results from both cohorts are reported as combined data in this report. For the closed-ended items, the 
percentage of respondents selecting each response option is reported. Note that a few of the percentage totals 
do not add up to 100% due to rounding error. In addition, because every respondent did not answer each item, 
the number of respondents answering each item is reported. The responses to open-ended items were coded 
and analyzed, and themes are reported. In some instances, the information provided by respondents fit into 
multiple themes; thus, the number of responses does not match the number of respondents. Chosen quotes are 
representative of the responses and are presented in italics. 


Results 


Demographic Information 

On both the pre- and post-surveys, respondents were asked to indicate their current position. Table 1 shows the 
position held by the 45 respondents at the time of the pre- and post-surveys. As Table 1 illustrates, most of the 
45 respondents served in a principal or assistant principal role at the time of the pre-survey and that was still the 
case on the post-survey (e.g., 83% were principals or assistant principals at the time of the pre-survey and 89% 
were in those roles at the time of the post-survey). On the post-survey, 38% of the respondents indicated that 
they have been in their current position for more than 5 years; 56% for 1-5 years; and 7% for less than 1 year. 
Three of the 45 respondents indicated a change in job position since the start of the program. 


Table 1. Current Position at Time of Pre-Survey and Post-Survey 


poehion Pre-Survey Post-Survey 

N % n % 
Principal 25 56% 27 60% 
Assistant Principal 12 27% 13 29% 
District-Level Department Director 3 7% 2 4% 
Superintendent - - - - 
Other® 5 11% 3 7% 
Total 45 100% 45 100% 


Respondents’ Perceptions of Their Expertise 

Respondents were asked to rate their level of expertise in several areas using a scale of basic, moderate, good, 
or highly developed (see Table 2). Note that none of the respondents rated their level of expertise as basic on 
the post-survey. The percent rating themselves as moderate decreased in every area from the pre- to post- 
survey; the percent rating themselves as good increased in every area from the pre- to post-survey; and the 
percent rating themselves as highly developed increased or stayed the same in all but one area from the pre-to 
post-survey (the percent rating themselves as highly developed in using student achievement data to make 
decisions decreased from 24% to 19% on the post-survey, although the percent rating themselves as good in this 
area increased from 43% to 62%). 


5 For example, other positions listed by respondents included positions such as Special Education Supervisor, Education 
Director, or Dean of Students. 
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Overall, areas in which participants rated their expertise as good increased for the most part due to moving from 
moderate to good rather than decreasing from highly developed to good. On the pre-survey, the majority of 
respondents rated their individual level of expertise as good in 4 of the 7 areas on the pre-survey; however, on 
the post-survey, the majority of respondents rated their expertise level as good in every area. The areas where 
respondents reported the greatest increase in expertise were: 
e developing teachers as leaders (27% increase from 35% good on the pre-survey to 62% good on the 
post-survey); 
e solving problems systematically by examining the whole picture, rather than isolated parts (25% 
increase from 39% to 64% good). 
e motivating teachers who are reluctant to consider new instructional practices (19% increase from 32% 
to 51% good); 
e generating enthusiasm for a shared vision for the school (11% increase from 17% to 28% highly 
developed); and 
e developing unity and teamwork among teachers (10% increase from 14% to 24% highly developed). 


Table 2. Level of Expertise Ratings 


: Pre Post Pre Post Pre Post Pre Post 
ae ne ads a% | - | 24% | 16% | 54% | 59% 14% 24% 
developing teacher leaders. (n=37) 5% - 41% 19% 35% 62% 19% 19% 
motivating teachers who are 
reluctant to consider new 8% - 49% 35% 32% 51% 11% 14% 
instructional practices. (n=37) 
generating enthusiasm for a shared ; ; 31% 11% 53% 61% 17% 28% 
vision for the school. (n=36) 

lyzi dint ti tudent 
ant fr eae ote : : 27% 16% | 49% | 57% 24% 27% 
sie he arene als cate - : 32% | 19% | 43% | 62% 24% 19% 
solving problems systematically by 
examining the whole picture, rather - - 44% 11% 39% 64% 17% 25% 
than isolated parts. (n=36) 


Respondents’ Perceptions of Their Work 

Agreement Indicators 

Respondents were asked to indicate their level of agreement using a scale of strongly disagree, disagree, agree, 
or strongly agree to statements about instructional practices (see Table 3), standards (see Table 4), and teacher 
performance and student outcomes (see Table 5). 


As Table 3 shows, most respondents on both the pre- and post-surveys agreed or strongly agreed with the 
statements regarding instructional practices. For all of the items, the percent who selected strongly agree 
increased on the post-survey (and the percent selecting disagree or agree decreased from pre- to post-survey); 
for example, respondents strongly agreed that they: 
e insist that the way to improve student learning is by improving instructional practices (22% increase 
from 24% to 46% strongly agree); 
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® encourage and support innovative improvement practices (17% increase from 32% to 49% strongly 
agree); 

e monitor the effectiveness of instructional practice in the school (11% increase from 19% to 30% strongly 
agree); and 

e challenge teachers to try innovative, evidence-based instructional approaches (10% increase from 29% 
to 39% strongly agree). 


On the pre-survey, approximately one-third of the respondents indicated they disagreed that they were able to 
motivate teachers who are reluctant to consider new instructional practices (33%), but that dropped to 14% on 
the post-survey (and the percent selecting agree to that statement increased from 56% to 72%). In addition, on 
the pre-survey, 19% disagreed with the statement that they insist that the way to improve student learning is by 
improving instructional practices and 17% disagreed that they challenge teachers to try new instructional 
approaches, but the percent disagreeing to both of those statements dropped to only 3% on the post-survey. 


Table 3. Level of Agreement with Statements Regarding Instructional Practices 
Strongly 
Item Statement Disagree 


Disagree Strongly Agree 


| monitor the effectiveness of instructional 


= = ‘0, 0, 9 9 0, 0, 
practice in the school. (n=37) oe ae 3% ne ec 
| insist that the way to Improve student learning ; ; 19% 3% 57% | 51% 2A% A6% 
is by improving instructional practices. (n=37) 
| challenge teachers to try innovative, evidence- ; : 17% 3% 58% | 58% 25% 39% 


based instructional approaches. (n=36) 
lam able to motivate teachers who are 
reluctant to consider new instructional - - 33% | 14% | 56% | 72% 11% 14% 
practices. (n=36) 

| encourage and support innovative 
improvement practices (such as coaching, 
modeling, observing practice, and providing 
feedback). (n=37) 


- = 3% 5% | 65% | 46% 32% 49% 


Table 4 illustrates agreement levels with statements regarding standards. As shown in the table, the percent of 
respondents selecting strongly disagree or disagree decreased for all items from the pre- to the post-survey. In 
addition, the percent of respondents selecting strongly agree stayed the same or increased for all items. The 
areas where respondents reported the greatest increase in agreement from the pre- to the post-survey were: 
e [convey to teachers how important it is for classroom assessments to be carefully aligned to content 
standards (25% increase in agreement from 67% to 92% agree/strongly agree); 
e | convey to teachers how important it is for classroom assessments to be carefully aligned to 
performance standards (20% increase from 64% to 84% agree/strongly agree); and 
e |am comfortable defining for teachers what constitutes effective standards for instructional practice 
(16% increase 84% to 100% agree/strongly agree). 
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Table 4. Level of Agreement with Statements Regarding Standards 
Strongly 
Item Statement Disagree 


Strongly 
Agree 
Post Pre _— Post 


Disagree Agree | 


lam comfortable defining for my teachers what 

constitutes effective standards for instructional - - 16% - 70% 68% 14% 32% 
practice. (n=37) 

| expect teachers to design lessons by working 


backward from the standards and assessments. - - 19% 8% 68% 68% 14% 24% 
(n=37) —_ | Ize 
| ensure that the school has a standards-based 

instructional system that is aligned with high - - 22% 11% 69% 64% 8% 25% 


performance standards. (n=36) 

lam clear in my expectation that teachers will meet 

regularly to assess student work against the - - 33% 22% 42% 53% 25% 25% 
_standards. (n=36) _ 


| convey to teachers how important it is for 
classroom assessments to be carefully aligned to 3% - 31% 8% 50% 61% 17% 31% 
content standards. (n=36) 


| convey to teachers how important it is for 
classroom assessments to be carefully aligned with 3% - 33% 17% 61% 53% 3% 31% 


performance standards. (n=36) 


Respondents were asked to indicate their level of agreement with statements about teacher performance and 
student outcomes (see Table 5). There was an increase in the level of agreement (i.e., the percent selecting 
agree or strongly agree) for all items from the pre- to post-survey. The following six statements showed 
increases in the percent agreement from the pre- to post-survey ranging from 11% to 31%: 

e | discuss why student depth of understanding is critical to achieving our mission (31% increase from 55% 
to 86% agree/strongly agree); 

e | search widely to benchmark our school’s performance to identify strategies that have enabled high 
performing schools to sustain improvement (21% increase from 41% to 62% agree/strongly agree); 

e | have institutionalized processes that ensure the professional development system is aligned with a 
belief that all students can achieve the same high standards (18% increase from 65% to 83% 
agree/strongly agree); 

e | have established a clear, written statement of learning and teaching principles for teachers to guide 
their improvement efforts (17% increase from 43% to 60% agree/strongly agree); 

e | ensure that school systems are successful in getting struggling students back on track (16% increase 
from 65% to 81% agree/strongly agree); and 

e | make judgments about teacher performance using a variety of data sources, including student results 
(11% increase from 83% to 94% agree/strongly agree). 


There were two statements in which the level of agreement showed a smaller increase: 
e | have strategies in place to buffer teachers from distraction to their instruction (6% increase from 70% 
to 76% agree/strongly agree); and 
e | have established an effective data management system to collect student performance data 
continuously throughout the school year (5% increase from 73% to 78% agree/strongly agree). 
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There was only one statement to which one respondent strongly disagreed (“I have established an effective data 
management system to collect student performance data continuously throughout the school year”), and the 
percent of respondents selecting disagree decreased for all items. The two statements with the highest level of 
disagreement on the post-survey each had approximately 40% of respondents reporting a level of disagreement: 
e | have established a clear, written statement of learning and teaching principles for teachers to guide 
their improvement efforts (41% disagree); and 
e | search widely to benchmark our school’s performance to identify strategies that have enabled high 
performing schools to sustain improvement (38% disagree). 


Table 5. Level of Agreement with Statements Regarding Teacher Performance and Student Outcomes 
Strongly Strongly 
Item Statement Disagree Agree 
Pre Post Pre Post Pre Post Pre Post 


Disagree Agree 


| have established a clear, written statement of 
learning and teaching principles for teachers to - - 57% 41% 35% 57% 8% 3% 
guide their improvement efforts. (n=37) 


| have established an effective data management 
system to collect student performance data 3% 3% 24% 19% 68% 51% 5% 27% 
continuously throughout the school year. (n=37) 
| ensure that our school's safety nets and 
interventions are successful in getting struggling 3% - 32% 19% 57% 59% 8% 22% 
students back on track. (n=37) 

| discuss why student depth of understanding is 
critical to achieving our vision. (n=36) 

| search widely to benchmark our school's 
performance to identify strategies that have 
enabled high-performing schools to sustain 
improvement. (n=37) 

| make judgments about teacher performance 
using a variety of data sources, including student - - 16% 5% 59% 62% 24% 32% 
results. (n=37) 

In my school, | have institutionalized processes 
that ensure the professional development system 
is aligned with a belief that all students can 
achieve the same high standards. (n=37) 

| have strategies in place to buffer teachers from 
distractions to their instruction. (n=37) 


- C 44% 14% 47% 69% 8% 17% 


11% = 49% 38% 38% 43% 3% 19% 


5 z 35% 16% 57% 59% 8% 24% 


3% : 27% 24% 65% 59% 5% 16% 


Frequency Indicators 

Academy participants were asked about the frequency of their interaction with teachers. As shown in Table 6, 
respondents reported an increase in frequency for all of the interactions asked about on the survey with the 
exception of one. The one interaction that over one-third (35%) of respondents consistently reported they do 
never or seldom was “model instructional strategies for teachers.” 
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On the post-survey, with the exception of the item noted above, more than half of the respondents reported 
their frequency for the other interactions as often or almost always. Over three-fourths of respondents said they 
do the following often or almost always: 

e discuss instructional issues with teachers (62% often, 14% almost always); 

e visit classrooms to briefly observe instruction (51% often, 27% almost always); 

e give teachers specific ideas for how to improve their instruction (75% often, 8% almost always); 

e use research evidence in decision-making (57% often, 22% almost always); and 

e Provide or locate resources to help staff improve their teaching (59% often, 16% almost always). 


Table 6. Frequency of Interaction with Teachers 
Never Seldom Sometimes Often Almost Always 


How often do you: 


Post Pre Post Pre Post Pre Post Pre Post 
discuss instructional issues : ; 3% ; 32% 2A% 65% 62% ; 14% 
with your teachers? (n=37) 
discuss equity issues with your 5% ; 22% 16% 6% 30% 27% 32% ; 22% 


teachers? (n=37) 


visit classrooms to briefly 


: : - - - 5% 16% 16% 68% 51% 16% 27% 
observe instruction? (n=37) 


watch an entire lesson when 
observing classroom - - 8% 6% 42% 36% 33% 28% 17% 31% 
instruction? (n=36) 


model instructional strategies 


8% 5% 27% 30% 46% 30% 16% 30% 3% 5% 
for teachers? (n=37) 


give teachers specific ideas for 
how to improve their - - - - 39% 17% 53% 75% 8% 8% 
instruction? (n=36) 


ask teachers about their use of 
data in instructional decision - - 14% 11% 38% 32% 41% 51% 8% 5% 
making? (n=37) 

make judgments about teacher 
performance based on student- 3% 3% 19% - 51% 41% 24% 54% 3% 3% 
level data? (n=37) 


use research evidence in your 


decision making? (n=37) - = 5% = 41% 22% 51% 57% 3% 22% 
ings? 

pee Teek er PEC IER tnes! 5% 3% | 3% | 5% | 22% | 22% | 49% | 49% | 22% | 22% 

provide or locate resources to 

help staff improve their - - 3% - 46% 24% 35% 59% 16% 16% 


teaching?’ (n=37) 


Table 7 shows the activities asked about on the survey that were more likely to happen annually or several times 
a year, rather than on a regular basis. Responses did not vary greatly from the pre- to the post-survey on these 
survey items. There was a decrease in how often teachers observe each other’s classrooms (a greater percent of 
the respondents on the post-survey indicated that the visits happen 3-5 times a year or annually). There was 


’ This item had slightly different wording in the pre-survey (“How often do you locate resources to help staff improve their 
teaching?”). 


Center for Applied Research and Educational Improvement, University of Minnesota 12 


also a slight shift from annually to 3-5 times a year on how often respondents communicate to the community 
about their commitment to high performance standards for all students, suggesting that more of the 
participants are doing this a little more often than before participating in the Academy. 


Table 7. Frequency of Engagement with the Community, Use of Data, and Peer Observations 


Greater than 


Monthl 3-5 times a year Annuall 
Item Statement Monthly y uf y 


How often do you communicate to the 
community your commitment to the 
achievement of high performance standards 
by all students? (n=37) 

How often do you rely on systematically 
collected evaluation data about your school 5% 8% 32% 32% 43% 41% 19% 19% 
in making decisions? (n=37) 


5% 3% 14% 16% 51% 59% 30% 22% 


How often do teachers observe each other's 


6% 3% 14% 8% 33% 36% 47% 53% 
classrooms? (n=36) 


Respondents’ Perceptions of Their Districts 

Academy participants were asked a series of questions about practices in their districts (see Table 8). For seven 
of the eleven items, the percent of respondents selecting strongly disagree increased, although this increase 
only represents a few of the respondents (e.g., 4% selecting strongly disagree represents only 2 of the 
respondents). For all but three of the eight items, the percent selecting disagree decreased. 


Although overall agreement (i.e., the percent selecting agree or strongly agree) went up on seven of the eleven 
items, agreement decreased on four of the items from the pre- to the post-survey. The items with the greatest 
change of agreement level (both increases and decreases) from the pre- to post-survey were: 
e Our district culture is one in which all teachers and administrators feel responsible for working together 
to improve student achievement (16% increase from 62% to 78% agree/strongly agree); 
e Our district offers a range of differentiated safety nets or interventions available at the individual 
student level (11% increase from 62% to 73% agree/strongly agree); 
e = Instructional leadership is a key component of our principal evaluation system (13% decrease from 91% 
to 78% agree/strongly agree); and 
e Our district has strategies to support principals and teachers in preparing students for college or other 
post-secondary education beyond high school (15% decrease from 87% to 72% agree/strongly agree). 
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Table 8. Level of Agreement with District-Level Practices 


Strongly Strongly 
Item Statement Disagree Agree 
Pre Post Pre Post Pre Post Pre Post 


Disagree Agree 


District staff help to ensure that our schools offer an aligned 
curriculum. (n=45) 


2% 2% | 16% | 13% | 60% | 73% | 22% | 11% 


Instructional leadership is a key component of our principal 


; 2% 2% 7% | 20% | 51% | 47% | 40% | 31% 
evaluation system. (n=45) 


District leaders often discuss instructional issues with school 


- 79 299 189 449 y 279 229 
leadership teams. (n=45) is a oe * i é A 


Our district offers a range of differentiated safety nets or 


0, 9 9 9 9 0, 9 9 
interventions available at the individual student level. (n=45) an as 36% [a >! a 1% Bee 


Our district has strategies to support principals and teachers 
in preparing students for college or other post-secondary - 4% 13% | 24% | 71% | 56% | 16% | 16% 
education beyond high school. (n=45) 


District administrators model high levels of professional 


; - 4% 7% 11% | 58% | 53% | 36% | 31% 
practice. (n=45) 


Our district culture is one in which all teachers and 
administrators feel responsible for working together to 2% - 36% | 22% | 38% | 56% | 24% | 22% 
improve student achievement. (n=45) 


Our district has institutionalized processes that ensure the 
professional development program is aligned with a belief - 4% 38% | 36% | 49% | 42% | 13% | 18% 
that all students can achieve the same high standards. (n=45) 


Principals participate in ongoing high-quality professional 
development. (n=45) 
Teaching is aligned to rigorous performance standards. 
(n=44) 

Our district supports equity-related initiatives. (n=45) - - 22% | 16% | 60% | 67% | 18% | 18% 


4% 2% | 18% | 11% | 47% | 53% | 31% | 33% 


o . 34% | 27% | 59% | 70% | 7% 2% 


Respondents’ Perceptions of Their Schools 
As illustrated in Table 9, the respondents’ agreement levels (i.e., the percent who selected agree or strongly 
agree) to these items increased from the pre- to the post-survey, except for two statements for which 
agreement levels decreased very slightly (2% and 3%, respectively) (“Administrators and teachers collectively 
plan who will provide leadership for initiatives” and “The schedule allows for adequate embedded time for 
collaborative teacher planning”). The statements for which agreement increased the most were as follows: 
e We provide an aligned curriculum for students across the grades (38% increase from 43% to 81% 
agree/strongly agree); 
e There is a strong commitment in this school to a common set of shared goals (19% increase from 70% to 
89% agree/strongly agree); 
e Our school improvement plan drives teachers' professional development (14% increase from 65% to 
79% agree/strongly agree). 
e Inour school, we have a range of differentiated safety nets or interventions available at the individual 
student level (14% increase from 62% to 76% agree/strongly agree); and 
e Team leadership at all levels focuses on the guidance, direction, and support of sustained improvement 
in instructional practice and student learning (14% increase from 67% to 81% agree/strongly agree). 
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Table 9. Level of Agreement with School-Level Practices 
Strongly 
Item Statement Disagree 


Disagree Strongly Agree 


We provide an aligned curriculum for students 
across the grades.® (n=37) 


3% 3% 54% 16% | 43% | 65% - 16% 


There is a strong commitment in this school to a 
common set of shared goals. (n=37) 

Our school improvement plan drives teachers' 
professional development. (n=37) 


3% ¢ 27% 11% | 51% | 65% | 19% 24% 


5% 3% 30% 19% | 38% | 38% | 27% 41% 


Administrators and teachers collectively plan who 


9 9 9 9 0, 9 0, 0, 
will provide leadership for initiatives. (n=37) gE 2% aoe as 37% am 19% 21% 


The schedule allows for adequate embedded time 


‘0, 9 9 9 9 9 9 10, 
for collaborative teacher planning.? (n=37) a ue one “> a Ce 208 


In our school, we have a range of differentiated 
safety nets or interventions available at the 3% - 35% 24% | 46% | 62% | 16% 14% 
individual student level. (n=37) 


Most of the parents of students in this school 
expect their children to go on to college. ?° (n=37) 
Team leadership at all levels focuses on the 
guidance, direction, and support of sustained 
improvement in instructional practice and student 
learning. (n=37) 


5% . 22% 24% | 57% | 57% | 16% 19% 


- . 32% 19% | 59% | 73% | 8% 8% 


Respondents’ Perceptions of Their Teachers 
As Table 10 illustrates, the majority of respondents agreed or strongly agreed with the statements regarding 
teacher practices (i.e., all items had over 60% agreement). There was an increase in the level of agreement with 
all but one of the six statements in this area. The percentage of respondents in agreement (81%) with the 
statement, “Teachers in this school have ongoing conversations among themselves about instructional 
practices,” remained the same from the pre- to the post-survey. The two statements where agreement levels 
increased the most were: 
e Teachers in this school have a sense of collective responsibility for student learning (14% increase from 
73% to 87% agree/strongly agree); and 
e Teachers in this school view problems as issues to be solved, not as barriers to action (12% increase 
from 52% to 64% agree/strongly agree). 


8 This item had slightly different wording in the pre-survey (“Our curriculum is vertically aligned across all grade levels.”). 

° This item had slightly different wording in the pre-survey (“The weekly schedule allows for adequate embedded time for 
collaborative teacher planning.”). 

10 This item had slightly different wording in the pre-survey (“Most of the parents of students in this school expect their 
children to go on to college or some form of post-secondary education beyond high school.”). 
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Table 10. Level of Agreement Regarding Teacher Practices 


Item Statement 


Teachers in this school are able to 
accurately monitor the progress of their 
students. (n=37) 


Strongly Disagree 


Disagree 


Strongly Agree 


14% 


5% 


76% 


65% 


11% 


27% 


Teachers in this school have a sense of 
collective responsibility for student learning. 
(n=37) 

Teachers in this school have the knowledge 
and skills they need to improve student 
learning. (n=37) 


3% 


3% 


24% 


19% 


14% 


8% 


59% 


70% 


57% 


76% 


14% 


11% 


30% 


14% 


Teachers in this school have ongoing 
conversations among themselves about 
instructional practices. (n=37) 


3% 


3% 


16% 


16% 


76% 


76% 


5% 


5% 


Teachers in this school prepare all students 
to go on to college." (n=37) 


32% 


30% 


68% 


57% 


14% 


Teachers in this school view problems as 
issues to be solved, not as barriers to action. 
(n=37) 


5% 


43% 


35% 


49% 


59% 


3% 


5% 


Respondents’ Time Spent on Instructional Leadership, Organizational Management, 


and Public Engagement 


Respondents were asked to report what percentage of their time in an average week they spend on the 
following activities: instructional leadership, organizational management, and public engagement. Table 11 
illustrates the percentage of time that respondents reported spending on each activity on both the pre- and 
post-survey. As Table 11 shows, the percentage of time spent in each area stayed about the same according to 
respondents; however, there is a large amount of variability in the percent time spent in these activities. For 
example, the range for the estimated percent of time spent on instructional leadership on the pre-survey was 
from 10% to 85% and on the post-survey the range was from 10% to 70%. The amount of time that respondents 
estimated they spend on organizational management and public engagement also had large ranges; thus, the 
mean percent time spent in these activities may not provide the clearest picture of how individuals are spending 


time in their roles. 


Table 11. Estimated Time Spent on Instructional Leadership, Organizational Management, and Public 


Engagement Activities 


In an average week, amount of time 


Pre-Survey Range 


Post-Survey Range 


Mean % Time Spent 


on the Activity 


eon Minimum Maximum | Minimum | Maximum Pre Post 
Instructional Leadership (n=37) 10% 85% 10% 70% 39% 37% 
Public Engagement (n=37) 5% 40% 0% 60% 11% 15% 
Organizational Management (n=37) 10% 85% 15% 85% 50% 48% 


1 This item had slightly different wording in the pre-survey (“Teachers in this school prepare all students for college or 
some form of post-secondary education beyond high school.”) 
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Respondents’ Recommendation of the Minnesota Principals Academy 

Respondents were asked to consider whether or not they would recommend the MPA to a colleague. The 
response options were “yes, why?” or “no, why not?” All of the respondents on the post-survey responded 
“yes.” For this section, comments from both cohorts and all respondents (i.e., not just those who had completed 
both the pre- and post-survey) were analyzed and are reported here. Thirty-nine (39) respondents provided 
comments. The comments centered around four main themes: (1) respondents felt the Academy provided high 
quality professional development; (2) respondents appreciated the opportunities for networking with people 
from other school districts and positions; (3) the respondents valued the learning around instructional 
leadership; and (4) the respondents found the new information and resources useful. The following quote from 
a respondent summed up three of the four themes: 


e High quality and ongoing professional development. The topics explored are relevant, timely, and in- 
depth. Ongoing interaction with fellow administrators in other settings and administrative leaders is 
invaluable. 


Most of the respondents (n=16) commented about the quality of the professional development. Representative 
quotes include: 

e Top quality professional development opportunity. Has the ability to transform your school/district. 

e The commitment to professional development is transformational. 

e It is the best, most comprehensive professional development I've ever had. The work challenged my 
brain and my practice and | was always excited about growing and pushing myself to increase 
competence. 

e = It is by far the most comprehensive training available to principals and leaders. 

e The best principal professional development that | have received. 


Eleven respondents mentioned networking as a primary reason that they would recommend the MPA. Exemplar 
comments include: 
e It isa good opportunity to network with other principals and to get good information on ways we can 
help students and teachers. 
e | feel the opportunity to network and share with the other principals in this area is critical. 
e It is a fantastic way to develop relationships with other principal's. It has also been the best 
professional development that I’ve ever received. 
e You are able to collaborate with professionals from other districts. 


Six of the respondents mentioned the emphasis on instructional leadership as being a key reason they would 
recommend the Academy to others. Comments in this area include: 

e It has helped me have a better understanding of what is required for strong instructional leadership 
and given me extended opportunities to collaborate with my peers. 

e The Principals Academy offers strong leadership for instructional learning. The cohort offers a wide 
array of professionals [coming] together.... The timing [of] principals getting together for a few days 
during the school year helps them to refocus their energy back on their school goals. We have learned 
what Great Principals need to do to create equitable schools for all children. 

e 2years of dedicated learning around me [as] an instructional leader and scholarly leader. 
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Finally, six respondents also mentioned the resources and materials as being a positive aspect of the Academy. 
Some comments include: 


The units are all excellent in guiding best practice for principals. 

The resources, time, content of the material. 

Provides current research on highly effective systems. You are able to collaborate with professionals 
from other districts. 

It gave me a broad base of new information to use in this position. Being with a cohort of like 
professionals was a great benefit. 


Respondents’ Comments about the Minnesota Principals Academy 
On the post-survey, participants were asked to respond to the following questions regarding their thoughts 
about the MPA: 


1. 


What is your greatest “take-away” from your participation in the Minnesota Principals Academy? 
(Please name /discuss only one.) 

What content (unit or otherwise) did you find to be the most helpful to you, and why? 

What are you doing differently now, if anything, in your school as a result of this experience? How do 
you see it impacting teachers and/or students? 

What feedback would you provide to improve the Minnesota Principals Academy? 

Do you have any additional comments you would like to share?” 


The comments provided in this report are from the post-surveys only, as the comments from the pre-survey 
were included in the previous evaluation report. In addition, for this section, comments from both cohorts and 
all respondents (i.e., not just those who had completed both the pre- and post-survey) were analyzed and are 
reported here. 


Greatest “take-away” 

The first open-ended question asked respondents to discuss their greatest take-away as a result of participating 
in the MPA. Forty-four respondents provided one or more comments to this question. The themes that surfaced 
when analyzing the comments provided in this section are similar to the themes discussed above regarding the 
reasons participants would recommend the MPA. 


The most common response from participants focused on the importance of leadership. Seventeen participants 
discussed the importance of being an instructional leader and possessing strong leadership skills. Some 
respondents talked about learning new ways to help struggling readers or teach mathematics; others explained 
it like this: 


Principals are the Key Instructional Leader: Your buildings are only as good as you are at leading them 
into a process of shared accountability and best practice. 

Effective school leadership is scholarly, data-based, and visionary. 

For me it is the importance of being an academic leader. To push myself to be aware of research and 
data, to engage staff in discussions about best practice and research. To offer staff opportunities to 
feed their own practice and brains. To nurture staff and teacher creativity and progress towards 
equity. 

Leadership only succeeds if the leader brings other people to the same vision and they are all able to 
work together and trust one another. The leader must be committed to continuous improvement and 
have continual support to foster a school environment of collaborative leadership ...The leader must 
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create alongside their team a strong vision for academic success for all students. The leader must work 
from strong ethical standards and stand for what their students need. The leader must improve 
instruction daily through strong feedback to teachers... 


Respondents also greatly valued the networking and collaboration opportunities they had with other colleagues 
(n=12 responses). Participants mentioned learning with, and from, other colleagues as being an important part 
of the program. Exemplar comments include: 
e The amazing network of colleagues and friends | have made over the last two years. The ability to 
synthesize the learning with a broad array of very intelligent school leaders. | feel "not alone" anymore. 
.. having a network now is an incredible resource for me. 
e Collaboration with colleagues. Time to share and learn what others are doing. 
e Besides all of the great professional development, the networking with other principals has been 
outstanding. 


Additional common responses from participants about their greatest take-away focused on the following: 
knowledge of research and evidence-based practices (n=5), the importance of having a strategic plan or vision 
(n=5), and understanding how to implement standards-based instruction (n=4). The quotes below represent 
these themes: 

e There is so much information | can use with our staff regarding evidence-based practices to promote 
students’ post-secondary success. This information has been valuable during the two years of 
Principals' Academy, and there will be many opportunities to use this information in the coming years 
as well. 

e Having a collection of materials and notes combined with the conversation around a specific example 
has given me views that | didn't have before. Using the research to make decisions makes more sense 
than what | feel needs to be done as a leader. 

e Having a clear vision is one of the most important facets for a school. From that, all decisions can be 
made. 

e [Greatest take-away was] Developing a road map and a goal for a school/district. 

e The importance of developing a standards-based instructional system to ensure students receive a 
viable curriculum. 


A few respondents also mentioned that they enjoyed the ongoing professional development opportunity of the 
Academy, seeing the connections between student expectations and equity, the importance of strategic 
decision-making, and the usefulness of having effective teams. For example, regarding decision-making, one 
respondent stated that their greatest take-away was “The importance of making strategic decisions using 
problem analysis, research, and creating a coalition of support.” 


Most Helpful Content 

The next open-ended question asked participants to name the most helpful content provided in the Academy 
workshops. Ten of the 42 respondents who answered this question listed either multiple topic areas or said they 
could not select just one. For example, one respondent said, “They were all good. It’s hard to focus on one...” 
Another said “all of them” and another said “picking one is very difficult.” 1 


12 All responses were analyzed, even if the respondent listed more than one area. 
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Eight of the respondents found one or more of the NISL units focusing on the specific curriculum elements of 
literacy, math, science, and/or social studies to be beneficial. Representative comments include: 

e The content that | enjoyed the most was in the content areas. Reading, Math, Social and Science. It 
was great to see the big picture and how to use the information. 

e The content that most resonated with me were the curriculum units. | came from an instructional 
background and having the research on effective learning and content standards grounded in a 
leadership lens made me solidify my vision as a leader—I am much stronger working with teachers and 
collaborating with how we will best impact student achievement. 


Seven of the participants mentioned the topic of “how people learn” as being very impactful. Two 
representative comments include: 
e | loved, loved, loved the How People Learn. | think it is instrumental to helping teachers in my school 
identify their most important job—to help kids make meaning of the content in impactful ways. 
e The unit that we discussed [about] how humans learn and the implications for Math and Literacy 
instruction. 


Seven respondents also listed the content around principals as drivers of change as the most helpful content 
from the Academy. Exemplar comments include: 
e = /really like unit 11, The Principal, driver of change. Very useful tools that | can use right now as | look 
towards a new schedule for my building. 
e The information on being a leader of change. 


Six respondents listed the content around strategic thinking as the most helpful; five listed coaching; and five 
listed the content around leadership as the most helpful. Representative quotes include: 

e The content around strategic thinking - from our second unit throughout the Academy - made the most 
impact. It allowed me to develop plans with other principals and walk away with both a plan and a 
mindset. 

e Coaching. It gave me great strategies for working with teachers to help build their skills as educators in 
a non-evaluative way. 

e Principal as instructional leader and team builder. | have had very little experience with effective teams 
in my current position. | am looking forward to developing those teams in my district and creating a 
much more powerful change tool! to shape our learning environment. 


Three or fewer of the respondents listed the following content as being the most helpful to them: content 
around equity, standards-based instructional systems, multi-tiered systems of support, professional 
development for teachers, the case studies, the accelerate work, and information on building capacity. 


Change in Respondents’ Practice 

Respondents were asked about what they were doing differently as a result of participating in the MPA. Thirty- 
nine respondents provided one or more comments to this question. Overwhelmingly, respondents talked 
broadly about changes in their leadership practice (e.g., restructuring leadership team meetings, driving change 
within their buildings) and taking on the role of instructional leader within their schools (n=21). Representative 
comments include: 
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e | have a much clearer idea of leadership best practices and how to implement them in everyday 
settings. 

e More visible leader. Not afraid to stand and deliver what is best for the school and students. 

e !look very differently at my daily work and discuss whether it falls into the arena of leadership or 
management. 

e !amworking with the teachers more. Before | was more about management. 

e lalso feel that |!ama stronger Instructional leader and can go into a classroom and [have an] 
understanding if good instruction is taking place. 

e The instructional leadership role of the principalship is more evident in the school, impacting teachers 
directly as planning and learning is taking place. |!am more strategic in planning out the work, based 
on clearly analyzing the needs of staff and student achievement. 

e Speaking to teachers frequently about quality teaching and what it looks like. More importantly what 
it does not look like. Our staff is beginning to see the issues with student success more clearly and 
talking more often about what needs to change. 


Included in this category is one respondent in a district leadership position who talked about supporting 
principals in the instructional leadership role. This person said, “At a district level, the need to keep supporting 
our principals and keep bringing the learnings from the academy to the table are critical. As a district our 
principals deserve feedback around instructional leadership more than anything else...” 


Eight respondents talked about an increased use of research and data in their practice. One respondent said 
they are “using much more research to support decision making and to guide programming.” Other comments 
include: 
e !am focused on research best practice to ensure we are meeting the needs of all students. The 
Academy has helped me make time in my schedule to read professional journals and books. 
e Looking at data more effectively. Working with an Instructional leadership team with a better focus. 


Five respondents mentioned having a vision for their school; for example, one discussed being “more intentional 
in creating a clear vision for our school and focusing time and professional development on that vision to 
increase student learning.” Other changes were mentioned by a few respondents as well, such as introducing 
power standards, aligning professional development for staff with the building improvement plan, reaching out 
to other principals and cohort members as resources, and looking at the processes and systems in place and 
making changes as necessary. 


Another component of the question asked respondents to think about how the changes they are making in their 
practice might be impacting teachers and/or students. Respondents discussed the impact in the following ways: 


e One respondent said that having a “full toolbox” from participating in the Academy impacts his/her 
teachers (“My teachers work reflects the same things because | model, set the expectations and share”). 

e Another respondent said that by using a systems lens, he/she is “now able to break down and tackle 
larger initiatives with the skills | gained from the Academy.” 

e One person said that, after participating in the Academy, he/she better understands the importance of 
having a vision statement; “...as a result, my school has gone through the process and written a vision 
statement.” 
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e By talking about teaching with the staff, “our staff is beginning to see the issues with student success 
more clearly and talking more often about what needs to change.” 

e By using research and data, “We have increased graduation rates, developed teachers instructional skills, 
hired more effective staff and created a vision that staff is committed to. The climate and culture of the 
school has significantly improved, students are attending college and taking more ownership in their 
education.” 


Feedback 

Participants were asked to provide any feedback that would help to improve the MPA. Thirty-six of the 
respondents provided one or more comments to this question. Much of the feedback provided by the 
respondents centered around these five themes: 

Positive comments about the program 

Meeting structure and activities 

Content and presenters 

Scheduling and timing of the program 

Action Learning Project 


oP WN 


Positive comments about the program. Nine respondents commented on the overall positive experience that 
they had in the Academy. Respondents said they enjoyed the networking and they feel the program should be 
offered throughout the state. Two exemplar quotes include: 
e 1|loved the academy. | feel that | would have never grown as much as a principal without the 
collaboration and the professional development | received while being in it. 
e This really has been a joy. |am grateful for the opportunity. 


Meeting structure and activities. Nine respondents also talked about the structure of, or activities during, the 
cohort meetings. Two respondents suggested providing more time to prepare for the meetings by laying out the 
homework in advance. Four respondents described the time they spent discussing their own experiences and 
challenges with other colleagues as valuable and saw this time as an important component of the meetings; 
they suggested that having even more time for discussion among colleagues would be beneficial. They indicated 
that talking with colleagues about issues that were more localized and present in their own schools was more 
powerful than some of the NISL content that may not have directly connected with what was happening in their 
schools. One respondent noticed that people tended to group themselves at the beginning of the program and 
suggested “more time for mixed collaboration for professional sharing.” Two respondents described how the 
different activities and ways of grouping people and gathering feedback was very helpful; one said that they 
would use them with staff members in their own building. 


Content and presenters. The content of the Academy meetings as well as the presenters was another area 
discussed by many participants. One person said they “found all the readings valuable, and felt it was a 
reasonable amount of work to expect from working principals.” Five participants shared various suggestions for 
improving the content which included: updating the materials, offering a unit on being a leader of technology, 
improving the EL unit, providing additional information around MTSS, having a more explicit focus on equity 
throughout the units, and providing a summary of each unit to refer back to easily. 


Participants also gave feedback about Academy presenters. Three respondents found the local presenters much 


more valuable than the NISL presenters, with the exception of the NISL math and science presenter. Another 
respondent suggested recruiting more presenters of color to participate in the program. 
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Scheduling and timing of the program. Seven respondents referred to the schedule and timing of the program. 
Three respondents shared that it was difficult to be gone from their building and suggested having 2-day 
sessions during the year and keeping 3-day sessions for the summer. One respondent wished the program could 
be three years instead of two years. One respondent found it difficult to have 4 days of content in June, and one 
respondent suggested avoiding meetings in July. Finally, one respondent shared that the gaps between meetings 
were helpful. 


Action Learning Project. Five participants discussed the Action Learning Project (ALP) as something that could be 
improved. Three participants suggested removing the Action Learning Project completely, with one of those 
participants suggesting a final presentation instead. Additionally, two participants suggested having more time 
before having to select their topic for the ALP. The participants described how having more time would allow 
exposure to more of the NISL units, which would make the project more productive and engaging. 


Additional Comments 
Respondents were also provided an opportunity to include any additional comments. Nineteen people 
responded to this item, and the majority commented on their appreciation for the Academy (n=13). Exemplar 
comments include: 
e |really enjoyed the academy and it really helped me grow as a professional. We never get the chance 
to have this kind of PD in Northwest MN! Thank you for bringing it! 
e Such a powerful experience. The principalship can be lonely and isolating; there is no better antidote 
for this than MPA. 
e = This is an invaluable experience and | applaud the MN Legislatures for funding the program. 
e Katie is doing an amazing job in this program. | am so thankful for her hard work and steadfast 
commitment to the academy. She has been truly supportive in my growth as a leader!!! 
e |really valued my time during the academy. In depth discussions around instructional leadership, case 
studies, and collegial conversations are rare. 


Three respondents commented on the speakers. One person said, “/ really appreciate the guest speakers.” The 
other two felt that some of the speakers were not as helpful as others and suggested a careful vetting system 
for speakers and possibly using local speakers who may be working in situations more similar to the participants’ 
contexts. 


Evaluation Question 2: Supervisor Feedback 


In addition to the surveys completed by participants, a separate online survey was sent to the supervisors of 
Academy participants. The survey was designed to provide information for Evaluation Question 2, "To what 
extent do the supervisors of Minnesota Principals Academy participants see a change in participants’ knowledge 
in the areas covered by the Minnesota Principals Academy (NISL) curriculum?" On the survey, supervisors were 
asked to report their level of agreement with a variety of statements relating to the participant’s (or, in some 
cases, participants’) practices in their school or district. 
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Methods 


In spring 2017, participants from both cohorts were asked to provide contact information for their current 
supervisor (in some cases, the supervisor was the district superintendent; in other cases, the supervisor was the 
building principal). There were cases in which multiple participants listed the same supervisor; however, each 
supervisor was asked to complete only one survey. The survey instructions were as follows: 


“When completing this survey, please think about the person in your school/district who you currently 
supervise and who is currently participating in the Minnesota Principals Academy. If multiple people who 
you supervise in your school or district are participating, please think in general about all of them when 
answering these survey items.” 


The 15-item survey (13 closed-ended items, 2 open-ended items) was sent to 33 supervisors in April 2017, some 
of whom supervised more than one participant. Of the 33 supervisors included, 25 had participants in the TC 
Cohort and eight had participants in the NW Cohort. A total of 17 (52%) supervisors responded to the survey (12 
from TC Cohort and 5 from NW Cohort). The 17 supervisors who responded represented 34 of the 53 
participants (20 of the NW Cohort participants and 14 of the TC Cohort participants). 


Results from both cohorts are reported as combined data here. For the closed-ended items, the percentage of 
respondents selecting each response option is reported. The responses to open-ended items were analyzed, and 
themes are reported. Chosen quotes are representative of the responses and are presented in italics. 


Results 


As illustrated in Table 12, over 90% of the respondents agreed or strongly agreed with all but one of the 12 
statements regarding Academy participation.” In that case, 12% (n=2) of the respondents disagreed with the 
statement, “The participant(s) regularly monitors and analyzes a range of student performance data to make 
judgments about areas of curriculum and instruction that need attention.” In addition, over 40% of the 
respondents strongly agreed with each of the statements, except for one item in which only 12% strongly 
agreed (“The participant(s) ensures that adequate resources are prioritized and allocated to the initiatives that 
will achieve maximum benefit in improving learning for all students.”) 


Table 12. Level of Agreement with Statements Related to Participation in the MPA (n = 17) 


Survey Item staal Disagree Agree ae J 
The participant(s) has enhanced his/her leadership skills. - - 41% 59% 
The participant(s) is able to motivate teachers or other staff 

members who are reluctant to consider new instructional - 6% 47% 47% 
practices. 

The participant(s) has acquired new knowledge and skills that 

allows them to think more systemically about how to improve - - 41% 59% 
instruction 

The participant(s) has demonstrated improved skills as a team 

builder (e.g., he/she promotes professional learning with staff " are aaie ane 


13 Each of the 17 respondents answered every item. 
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Strongly 
Agree 


Strongly 
Disagree 


Disagree Agree 


Survey Item 


members, shares leadership, supports ideas offered by team 
members). 

The participant(s) helps build a school/district-wide 
commitment to effort-based theories of learning where all - - 53% 47% 
students can learn—given time and support. 


The participant(s) recognizes the individual needs of his/her 


- 69 419 539 
staff members and supports them accordingly. a “ ‘ 


The participant(s) ensures that adequate resources are 
prioritized and allocated to the initiatives that will achieve - 6% 82% 12% 
maximum benefit in improving learning for all students. 


The participant(s) has a compelling school vision in which all 


- i) 0, 9 
students are held to high expectations. oe oa ae 


The participant(s) ensures that teachers use data to make 
instructional decisions for students. 

The participant(s) regularly monitors and analyzes a range of 
student performance data to make judgments about areas of - 12% 47% 41% 
curriculum and instruction that need attention. 


- 6% 41% 53% 


The participant(s) is committed to identifying research and best 


. ; ; . - 6% 53% 41% 
practices that support instructional improvement. 


The participant(s) is committed to listening to community 
members, advocates, or other stakeholders to inform change - - 59% 41% 
within the school. 


Supervisors were provided an opportunity to provide written feedback on the survey. They were first asked, “In 
what ways, if any, has participation by one or more of your colleagues in the Minnesota Principals Academy 
impacted your district or specific schools in your district?” Nine of the survey respondents provided comments 
to this question. Below are a few of the comments made by respondents regarding ways in which the participant 
has grown and is contributing to the district. 

e Our participant began as a leader among her peers and MPA has further strengthened her skills to do so. 
She has initiated and piloted new research-based strategies which have demonstrated strong results at 
her building which we now plan to expand into other sites, eventually all. 

e She has provided vision and direction towards tiered interventions and systems for managing student 
behaviors and academic achievement. 

e The MN Principal Academy has served as a great resource to our school. The learning experiences are 
well-aligned with the vision and the school. Most of all, | notice the intentionality of the participant with 
engaging staff, family and students in the learning process. There is a strong sense of increased passion 
to meet the needs of every student! 


In some instances, the respondents commented on ways in which the district is benefitting from the 
participation of school/district staff members in the Academy. 


e _... We had principals who prided themselves on managerial duties. The transformation the Principal 
Academy has given our district is principals who have developed a deep knowledge of understanding 
what leadership is and how it should look as a principal. We now have a common vision as a district of 
what instructional leadership is, how it looks, and how to implement it. We now understand that 
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teaching and learning must be at the top of the priority list on a consistent basis. Leadership is a balance 
of management and vision. While leaders cannot neglect other duties, teaching and learning should be 
the area where most of the leaders’ scheduled time is allocated... As a district ... It is important that we 
support their [principal’s] improvement in the area of instructional leadership now that we have a 
common vision and understanding. 

e__,.. |have seen a lot of growth in regards to how we choose to do things in our district. ... For example, 
each building has looked at their safety nets to determine if they are effective or not. This determination 
is not based upon what people like or want but instead on results that we are getting with the safety net. 
Those that are not functioning well are being phased out and other potential practices are being looked 
at. The latest unit, change, has been very helpful for us in navigating how we are going to implement a 
change in what we are going to teach, how it is going to be taught, and finally, how we will know if it is 
working or not..... 

e As adirect result of this learning experience, our ability as a building administrative team to make sense 
of data, and act on that data in ways that benefit student learning, has increased. We're far more likely 
to question and sit with uncertainty regarding achievement or behavioral data than we are to rush to 
judgement or conclusion. We're slowing down and thinking more.... We're thinking more deeply about 
how to develop and measure non-cognitive skills, and we are shaping professional development of staff 
around learning and brain research, especially as it pertains to culturally responsive teaching. 


Respondents were asked to provide additional comments about the MPA. Ten respondents provided comments. 
Four respondents discussed challenges. One stated that finding funding to continue to send people to the 
Academy is a barrier. Another commented on the Action Learning Projects of the two participants this person 
supervised. This supervisor said that one of the projects did not align well with district needs and the other 
project was unknown to the supervisor. Two supervisors discussed the challenge of the time commitment and 
principals being out of their buildings — one stated: 

e The biggest challenge | see in the future is the time commitment for participants. ... How can the 

Principal Academy offer more opportunities that do not conflict with the daily school calendar? 


Three respondents provided general comments indicating that they felt the MPA offers a strong curriculum, is of 
high quality, and is worth the investment. For example, two respondents stated: 
e The feedback on the MN Principals Academy has been overwhelmingly positive. Well worth the time our 
participant has spent away from the building. 
e _.,.. can still see that the new curriculum is better aligned to what leaders really need to do to make 
effective school improvements. 


Finally, three respondents provided examples of how the MPA is influencing work in their districts: 

e For our district, | firmly believe it has and will continue to make us be stronger and more effective. 
Principals are utilizing the concepts shared with us in the academy with staff in their buildings and | 
believe that we are on our way to changing our academic achievement from a flat line to a growth line. | 
have also noticed that the conversations we have as an admin team are more focused on topics that will 
help us improve rather than those topics that are more about complaining versus finding a solution. |am 
thrilled that we have been provided this opportunity and believe the long-term benefit for our district will 
be felt for many years to come. 

e |!cannot emphasize the importance of the Academy especially in outstate MN. It has always been [that] 
the bigger urban areas have had more access to highly effective professional development. As a district, 
the vision around instructional leadership the Academy has given us is invaluable. ... As a district leader 
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participating alongside our principals we have a clear vision of how to support our principals. [This 
respondent provided examples of how to support principals, such as “We need to provide our Principals 
with strong guidance on curricular and instructional improvements” and “Tools and processes that 
principals can use to ensure instruction is aligned with the district's goals and standards” and “A culture 
and support for the use of data beyond simple test scores to improve student outcomes.” | 

e | believe our recent experience redesigning a year-long course, in partnership with the teachers that lead 
it, is highly representative of the two years of learning that took place in the MN Principals Academy... 
The course is a problem-solution oriented, authentic literacy experience ... and the design of it touched 
on many themes that were part of the MN Principals Academy. They include: the offering of real-world 
learning experiences, integrating student choice and voice in their learning, an emphasis on non- 
cognitive skill development, leadership by partnership, standards-based learning, tapping the "right 
who's" to lead the learning, and creating parameters for measuring student learning and success. 


Finally, supervisors were also asked, “In the future, what is the likelihood that you would send additional 
members of your staff to participate in the MN Principals Academy?” Ninety-four percent of respondents 
reported they were likely or very likely to do so, with only one supervisor indicating unlikely. 


Summary 


The purpose of this evaluation was to describe and understand the MPA as a resource and training program for 
school and district leaders. The evaluation was designed to provide information regarding the extent to which 
participants gained knowledge and skills, or changed their behaviors, from the beginning of the program to the 
end of the program in the areas covered by the MPA/NISL curriculum. To do this, we administered a pre- and a 
post-survey to Academy participants. The majority of respondents to both the pre- and post-surveys were 
principals, although others also served in roles such as assistant principal, dean of students, special education 
supervisor, district-level department director, and so forth. At the end of the program, we also surveyed 
participants’ supervisors to ask them about changes they were seeing in participants’ knowledge, skills, or 
behaviors since participating in the Academy. In general, the results from both surveys on the closed-ended 
items suggest a positive change in participants’ knowledge, skills, and behaviors. In addition, the comments 
made by both participant and supervisor respondents about the program were overwhelmingly positive. In fact, 
all participant respondents said they would recommend the MPA to colleagues, and 94% of the supervisor 
respondents said were likely or very likely to send additional staff members to the MPA. 


Looking across all of the survey items and comments a main finding emerges — the Academy participants are 
thinking more about leadership and the role of an instructional leader. The participants are noticing this change 
and so are their supervisors. Participants now know more about that concept and are trying to determine how 
that role will fit within their jobs, their schools, and their districts. For example, although the percentage of time 
that respondents said they spend on instructional leadership did not change from the pre- to the post-survey, 
for all of the items about instructional practice, the percent who selected strongly agree increased on the post- 
survey. There was a 22% increase in the percentage of respondents selecting strongly agree regarding their 
belief that “the way to improve student learning is by improving instructional practices.” Academy participants 
were also asked to estimate the frequency of specific interactions with teachers related to being an instructional 
leader. Over three-fourths of respondents said they discuss instructional issues with their teachers, visit 
classrooms to briefly observe instruction, and give teachers specific ideas for how to improve their instruction 
often or almost always, and all of these ratings increased from the pre- to the post-survey. 
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In addition, respondents were asked to share comments about what they considered the greatest take-away 
from the Minnesota Principals Academy. The most common response from participants focused on the 
importance of leadership. Seventeen participants discussed the importance of being an instructional leader and 
possessing strong leadership skills. Many respondents also said that the most helpful content (if they had to pick 
just one) was learning more about instructional leadership and coaching in the content areas (math, science, 
literacy, and social studies). This seems to be an area that respondents are still working on since the one 
interaction that over one-third (35%) of respondents reported they do never or seldom with teachers is 
modeling instructional strategies for them. 


In keeping with the theme of instructional leadership, when respondents were asked to discuss how their 
practice had changed as a result of participation in the Academy, they overwhelmingly talked broadly about 
changes they are making already in their leadership practice (e.g., restructuring leadership team meetings, 
driving change within their buildings) and taking on the role of instructional leader within their schools. Thus, 
respondents understand the importance of the principal as the instructional leader in the school, but they have 
not yet had the time and experience to bring what they learned in the Academy back to their buildings. In the 
next few years, they will likely try to determine how this will look for them in their buildings and districts (e.g., 
some might decide to rearrange their own time so that they can provide more instructional coaching themselves 
while others might rearrange resources to hire the coaches necessary to provide their staff with modeling, and 
feedback on their, instructional practice). 


In summary, the feedback about the MPA from participants and supervisors was very positive. In addition, the 


results from the pre- and post-surveys suggest that participants are making changes in key areas identified by 
the MPA/NISL curriculum. 
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Appendix: Information on Retrospective Pre-Survey 


As part of the CARE! evaluation of the Minnesota Principals Academy, evaluators were asked to compare two 
survey designs in an attempt to determine the most efficient and accurate measure of participants’ perceptions. 
One design included obtaining information from participants at two data collection points: first, from a pre- 
survey administered in Fall 2015 and then from a post-survey administered after the completion of the program 
in April 2017. In the second design, Academy participants were asked to respond to a retrospective pre-survey. 
In the retrospective pre-survey design, survey respondents are asked to rate themselves before and after the 
program during a single data collection even (after the program or intervention has occurred). For example, 
during the administration of the retrospective pre-survey, Academy participants were asked to think about 
themselves (e.g., their beliefs, attitudes, behaviors) in that moment (i.e., "After the program") and to think 
about themselves before they began participating in the MN Principals Academy and then to indicate the extent 
of their agreement with each survey statement at these two different points in time. 


Background on Retrospective Pre-Survey Design. In order to assess the overall effectiveness of an intervention, 
a traditional pre- and post-survey design is often used to measure changes in participant behavior or knowledge. 
However, studies have found that when participants are asked to self-report at the beginning and at the end of 
the intervention, results may be misleading if participants use a different frame of reference to evaluate their 
own knowledge and skill on the post-survey than they used when filling out the pre-survey (Howard, Ralph, 
Gulanick, Maxwell, Nance, & Gerber, 1979; Klatt & Taylor-Powell, 2005). Howard and colleagues (Howard et al, 
1979; Howard, 1980) call this a “response shift bias” when participants have a more in-depth understanding of 
what is being measured as a result of participating in the program or intervention and therefore answer the 
questions differently than how they answered them on the pre-survey. In order to reduce response shift bias, 
researchers have recommended the use of a retrospective pre-survey design, in which participants rate 
themselves both before and after the intervention in a single data collection event that happens after the 
intervention has occurred. In order to determine whether response shift bias has occurred, Pratt, McGuigan & 
Katzev (2000) suggest comparing responses on the pre-survey to responses on the retrospective pre-survey. 


Analysis. After administering both survey designs to Academy participants, CAREI evaluators compared 
responses on the first pre-survey to those on the retrospective pre-survey using a nonparametric sign test. 


“The “‘paired-samples sign test’, typically referred to as just the ‘sign test’, is used to determine whether 
there is a median difference between paired or matched observations. The test can be considered as an 
alternative to the dependent t-test (also called the paired-samples t-test) or Wilcoxon signed-rank 

test when the distribution of differences between paired observations is neither normal nor symmetrical, 
respectively. Most commonly, participants are tested at two time points or under two different 
conditions on the same continuous dependent variable.” (SPSS, 2017) 


We found very few statistically significant differences between the pre-survey and retrospective pre-survey 
responses; in fact, only 5 out of 66 items were significantly different (suggesting little response shift bias). 
Because there was little evidence of response shift bias and the number of participants who completed the pre- 
and post-surveys was greater than the number who completed the retrospective pre-survey, the decision was 
made to use the pre- and post-survey results in this report. However, considering the literature on the 
retrospective pre-survey design and our findings, we would recommend further discussion about using the 
retrospective design in the future for ongoing Academy evaluations. 
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